Systems and methods for tracking impacts

ABSTRACT

Surface impacts are located and characterized based on an acoustic signal produced by the impact despite the presence of signal dispersion. Acoustic signals from the surface may be compared to acoustic signals detected external to the surface in order to eliminate spurious impact sensing due to external sounds. Low-frequency acoustic signals may be sensed and identified as explicit hard “bangs” which are of limited utility for pointing and tracking applications.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.10/155,874, filed May 24, 2002, now U.S. Pat. No. 7,643,015, the entiredisclosure of which is herein incorporated by reference.

FIELD OF THE TECHNOLOGY

The invention relates generally to the field of acoustic signalprocessing and, in particular, to acoustic signal processing forlocating surface impact coordinates.

BACKGROUND

Interest in converting inanimate surfaces such as walls, floors,windows, and the like into sensate surfaces (e.g., surfaces that canlocate and characterize object contact, impacts or proximity) hasrecently grown. The heightened interest is, in part, a result ofincreased attention to large-scale interfaces for human-computerinteraction in smart homes, public spaces, workplaces, etc. Generally, asensate surface is created with the addition of transducers on, adjacentto, or integral with the surface. The transducers track the positions ofobjects on or adjacent the surface.

Much development has gone into the production of “touch screens,” acommon example of an interactive surface that tracks the position offingers against a video monitor. The technologies in touch screens(resistive sandwiches, active acoustic absorption tracking, capacitivesensing, infrared LED-receiver grids, for example) generally do notscale gracefully to large interactive surfaces (e.g., of over a meter inlength), because of complexity and expense. Large video displays (e.g.,from video walls and projections) are already becoming common in publicspaces, but are not yet interactive, partly because of the difficultiesinherent in reliably and economically scaling the appropriatetechnologies for tracking hands across the active surface.

Impact-locating systems are one current area of sensate-surfacedevelopment. Some impact-locating systems identify the location of asurface impact by analyzing the resulting acoustic signal. This approachis attractive because of its simplicity, economy, and minimalinterference with the surface (a set of contact transducers may be gluedonto the surface to receive the acoustic wavefront from the impacts);people can simply knock or tap on the surface to interact. Glass is aubiquitous material used for large surfaces in present-day construction.Because glass is transparent, it can be used to dynamically displayinformation behind the surface, and is therefore an attractive materialfor use as a sensate surface. As all sensors can be mounted on theinside surface of a single-paned sheet of glass, it is well-suited forinstrumenting outside-facing windows; e.g., it forms a straightforward,secure, and practical means for making shop windows interactive forpassers-by outside. However, acoustic signal propagation in glass ischaracterized by dispersion, i.e., different frequencies propagating atdifferent velocities, creating the tendency of the signal to spread outas it travels along the glass surface.

The dispersive nature of acoustic signals in glass is problematic forimpact-locating systems because the time of arrival of the acousticsignal at each transducer location is difficult to determine. Forsignals originating from an impact on glass, dispersion is generallyseen as a variable amount of low-amplitude higher-frequency acousticsignals that often travel ahead of the main wavefront. The dispersionresults in an acoustic signal whose characteristics vary as the distancefrom the impact increases. Thus, the acoustic signal characteristicsreceived at each transducer will vary from one another unless thetransducers are precisely equidistant from the point of impact.Additionally, the frequency components created by an impact can vary,depending upon the object that strikes the surface, causing acorresponding variation in the primary acoustic propagation velocitythrough dispersion. Thus, a specific propagation velocity cannot beassumed when determining the impact coordinates unless the impact sourceis known or characterized.

In theory, the differential time of arrival of the acoustic signalbetween each of at least three transducers allows an impact location tobe determined. One such approach is described by Paradiso et al. in“Sensor Systems for Interactive Surfaces,” IBM Systems Journal, Vol. 39,Nos. 3 & 4 (2000). In accordance with the approach described therein,impact-location coordinates are mapped based upon the differentialtimings of the acoustic signal received at each of four transducers.However, because the differential time of arrival must be determined,the leading edge of the acoustic signal must be accurately identified sothat the signal arrival time at each transducer is detectedconsistently. In practice, the approach described in the above-mentionedarticle can fail to accurately locate the impact coordinates when signaldispersion prevents accurate identification at each transducer of boththe acoustic signal wavefront and the corresponding acoustic signal timeof arrival. Additionally, the preceding approach fails to detectlow-frequency acoustic signals created by certain impact sources, e.g.,a fist, and can be vulnerable to interference from ambient noises.

SUMMARY

Impact-locating is most effective when the acoustic-signal wavefront isaccurately identified despite signal dispersion. An improved approach tolocating impacts includes signal analysis that accurately identifies theacoustic signal wavefront and the corresponding acoustic signal time ofarrival even where the acoustic signal includes dispersion.

In one aspect, the invention locates an impact on a surface afterreceiving, at a plurality of locations, an acoustic signal created bythe impact by cross-correlating the received signals to determine anarrival time at each location.

In another aspect, the invention locates an impact on a surface afterreceiving an acoustic signal created by the impact, but ignores acousticsignals created by external sounds. In one embodiment, the inventionincludes an air transducer that detects acoustic signals created byexternal sounds. In a version of this embodiment, the air transducer isa crystal microphone.

In a further aspect of the invention, low-frequency acoustic signalscreated by surface impacts are detected at one or more locations. In oneembodiment, the invention detects low-frequency acoustic signals below50 hertz. In one version of this embodiment, the low-frequency acousticsignals are detected by an electrodynamic cartridge.

In still another aspect of the invention, an impact on a surface islocated by removing wave dispersion effects from the wavefront of anacoustic signal received at a plurality of locations. In one embodiment,the invention features an acoustical model of the surface. In anotherembodiment, an impact source is identified.

In still a further aspect of the invention, a graphical user interfaceis created on a surface by a combination comprising a plurality ofsensors that detect surface impacts, a data acquisition system thatdetermines impact locations on the surface using cross-correlation, andan image generator that responds to the data acquisition system. In oneembodiment, the invention includes a laser rangefinder to also track thecontinuous position of hands above the surface. In another embodiment,the invention includes a microwave or other rangefinder that can sensethrough the glass and respond to people and objects approaching.

In yet another aspect of the invention, an impact source ischaracterized. In one embodiment, a data acquisition system iscalibrated with known acoustic signal characteristics for at least oneimpact source, an acoustic signal created by an impact source isreceived at one or more locations, and the acoustic signal is analyzedwith the data acquisition system. In another embodiment, a wave velocityis determined.

BRIEF DESCRIPTION OF THE DRAWINGS

Other aspects, advantages and novel features of the invention willbecome more apparent from the following detailed description of theinvention when considered in conjunction with the accompanying drawingswherein:

FIGS. 1A and 1B are graphs of acoustic signals in glass, with FIG. 1Abeing an acoustic signal created by a knuckle tap and FIG. 1B being anacoustic signal created by a metal tap.

FIG. 2 is a block diagram of an impact-locating system in accordancewith the invention.

FIG. 3 is a flow chart of signal analysis performed on an acousticsignal in accordance with the invention.

FIG. 4 is a schematic of a pre-amplifier in accordance with theinvention.

FIGS. 5A and 5B are a schematic of an analog signal conditioner inaccordance with the invention.

DETAILED DESCRIPTION

Acoustic Signal Propagation. Acoustic signal propagation depends, inpart, on the acoustic properties of the material comprising the sensingsurface. Acoustic properties are those that affect acoustic signaltransmission through the surface material. For example, these propertiesmay affect either or both of the acoustic signal velocity and the signalattenuation. As a result, impact-locating systems and methods desirablyaccount for acoustic properties of the sensing surface.

Further, the impact source (e.g., knuckle tap, metal ring, etc.) alsoaffects the acoustic signal. A glass sensing surface provides oneexample. An lower frequency acoustic signal created when glass is struckwith a relatively soft impact source, such as one or more knuckles orfingertips, propagates more slowly than a higher-frequency acousticsignal created when a sharp impact is made on the glass surface, forexample, when the surface is struck with a rigid object. In glass, thetransmission speed of signals generated by sharp impacts may be up toten times greater than the transmission speed of signals generated bysoft impacts. For example, in a panel of safety glass 1 cm thick,propagation velocities of 1200 m/sec for knuckle taps and 2000 m/sec formetal taps were observed. Additionally, the propagation speed isaffected by both the thickness of the surface and the surface material.For example, propagation speeds observed in safety glass having athickness of 1 cm are roughly 40% faster than propagation speedsobserved in tempered glass having a thickness of 0.635 cm.

Sharp-impact and soft-impact sources are further distinguished by thenature of the acoustic signal that each generates. When a glass surfaceis struck with a relatively soft-impact source, both structural acousticsignals and audible acoustic signals are transmitted in the sensingsurface. The structural acoustic signals are referred to as flexuralwaves. A flexural wave is structural in nature because it propagates bydeflection within the material through which it travels. Flexural wavesinclude both high- and low-frequency components. They are difficult toanalyze because a flexural waveform spreads out as it propagates fromthe impact origin. This effect, referred to as dispersion, resultsbecause higher frequency flexural waves travel faster than lowerfrequency flexural waves.

FIGS. 1A and 1B illustrate the difference in acoustic signals generatedby different impact sources. FIG. 1A shows an acoustic signal resultingfrom a knuckle tap on a glass surface, whereas FIG. 1B shows an acousticsignal generated by striking a metal ring against a glass surface. Thewaveform of FIG. 1A demonstrates the effects of dispersion on theacoustic signal created by the knuckle tap. The dispersion results in asignal that includes a variable amount of low amplitude,higher-frequency components that often arrive before the lower-frequencysignal components. In FIG. 1A, the change in signal frequency is mostclearly seen as the signal decays. However, the effects of dispersionalso make it more difficult to determine when a particular acousticsignal has arrived at a given transducer location because the risingedge of the signal is less clearly defined.

As can be seen in FIG. 1B, sharp impacts propagate via an acoustic modethat maintains both a stiffer rising-edge and more uniform frequency,and is generally not distorted as much by dispersion. Thus, signaldetection is more straightforward for sharp impacts because the acousticsignal arrival time is more easily determined.

System Overview. Referring now to FIG. 2, an impact-locating system 10includes a plurality of transducers 12 _(1,) 12 _(2,) 12 _(3,) 12 ₄(collectively, transducers 12) located on a sensing surface 14. Onlythree transducers 12 are required to accurately locate impactcoordinates on a two dimensional sensing surface 14. The fourthtransducer 12, while not necessary, is preferred for redundancy anderror checking In the embodiment shown in FIG. 2, one transducer 12 islocated adjacent each of the four corners of sensing surface 14.However, the transducer location may be varied to include, for example,edge-centered orientations, circular orientations, asymmetricalorientations, associated transducer pairings, and the like so long asthe system is configured and calibrated to accommodate the selectedorientation. In one embodiment, the transducers 12 are located away fromthe edge of sensing surface 14 in order to reduce both signalattenuation and the effects of signal reflection.

In one embodiment, the transducers 12 are electret microphones. Inanother embodiment, the transducers 12 are made of polyvinylidenefluoride (PVDF). In still another embodiment, the transducers 12 arepiezoceramic transducers. In one version, the piezoceramic transducersdetect acoustic signals with a frequency of 300 Hz-5000 Hz. Thetransducers 12 may be attached directly to the sensing surface 14. Avariety of attachment means may be used including adhesives, fastenerssuch as nuts, bolts, and screws, and the like so long as the transducer12 is mechanically secured to the sensing surface 14 and is capable ofreceiving acoustic signals propagating therealong.

Output signals from each transducer 12 are supplied to a dataacquisition system 16. The transducer 12 output signal may be amplifiedwith a pre-amplifier 17 (FIG. 4) to increase the output signal strengthand improve the signal-to-noise ratio of the output signal. In oneembodiment, a separate pre-amplifier 17 is mounted with each transducer12. The pre-amplifiers 17 may also be shielded to reduce the effects ofelectromagnetic interference. In one version, the pre-amplifiers 17 areshielded with copper shielding.

In one embodiment, the data acquisition system 16 includes a signalconditioner 18, a digital signal processor 20, and a computer 22. Thedata acquisition system 16 continuously samples the output of eachtransducer 12. In the embodiment shown in FIG. 2, there are a total offour channels that are monitored, one channel for each transducer 12. Arolling buffer (not shown) in the data acquisition system 16, typicallyimplemented in a random-access memory of computer 22, receives theinputs from each channel in digital form. A signal conditioner 18schematic is shown in FIGS. 5A-B.

Advantageously, the digital signal processor 20 has a sampling ratesubstantially greater than the highest frequency signal detected bytransducers 12 in order to provide ample waveform detail for signalanalysis. In one embodiment, the sampling rate of the digital signalprocessor 20 is ten times greater than the highest frequency signaldetected by the transducers 12. Because serial input ports are commonlyavailable in computers 22, it is also advantageous for the digitalsignal processor 20 to be equipped with a serial output connection.

As is discussed in greater detail below, the data acquisition system 16is preferably calibrated to the acoustic properties of sensing surface14. Algorithms described below are used by the data acquisition system16 to determine the impact source and impact location P. The signalconditioner 18 filters the transducer outputs and converts them from ananalog format to a digital format, and the resulting digital data isreceived by computer 22 via a serial link. In one embodiment, thedigital signal processor 20 performs peak timing analysis while thecomputer 22 performs impact source characterization and impact locationestimation. The computer 22 may be a personal computer or any type ofprocessing system such as a microprocessor, that can be programmed toexecute the impact-location algorithms described herein.

The embodiment shown in FIG. 2 includes cable interconnections betweenthe various impact-locating system components. However, in anotherembodiment, the transducers 12 communicate with the data acquisitionsystem 16 via radio frequency (RF) or optical signals and cableinterconnections are not required.

Arrival of the acoustic signal at a transducer 12 is determined to occurwhen the acoustic signal exceeds a preset threshold. When a signal isdetected, its digital representation (generated by digital signalprocessor 20) is stored by computer 22 in a rolling buffer assigned tothat transducer 12. The threshold may be any signal level so long as itis above the noise level and can be unambiguously detected. In oneembodiment, the threshold is an acoustic waveform peak value determinedby a system designer. A triggering event occurs when the acoustic signalreceived from any one of the plurality of transducers 12 exceeds thethreshold. Triggering events cause the data acquisition system 16 tostore and analyze acoustic signal information from the transducers 12.In one embodiment, the acoustic signal information that is stored andanalyzed is received between a time just preceding the triggering eventand a time immediately following the triggering event. This data storageapproach is advantageous, in part, because the arrival time of theacoustic signal at each transducer 12 is likely to be different. Thus,this approach allows the acoustic signal arrival time to be determinedfor each transducer 12 regardless of whether the transducer 12 is thefirst to detect an acoustic signal created by a surface impact. Thisapproach also enables robust triggering on the higher amplitudesgenerated by an impact, while still digitizing and storing the muchlower amplitude signals that come beforehand, such as higher-frequencycomponents separated by dispersion.

In one embodiment, a dynamic threshold may reduce false triggers thatcan be created by peak amplitude magnification resulting from acousticsignal reflection and associated signal coupling between a signalcreated by a newly-occurring impact and a reflected or resonating signalfrom an earlier impact. In this embodiment, the threshold adjusts upwardimmediately after an impact is sensed and gradually returns to theoriginal threshold during a preset time period, provided that nointervening impacts occur. The system still responds to interveningimpacts (although with less sensitivity because of the higherthreshold), but is not falsely triggered by residual sound in the glassremaining from the previous impact.

In one embodiment, the digital signal processor 20 detects thetriggering event. In a another embodiment, a digital signal processor 20is not used and the computer 22 analyzes the acoustic signals fortriggering events. The acoustic signals from identical time periods areanalyzed for each transducer 12.

In one embodiment, the data acquisition system 16 continuously samplesthe signals from each transducer 12 and stores them, on a continuousbasis, in a corresponding rotating buffer. Upon detecting a transducersignal above a noise threshold, triggering event is declared, and datafrom each transducer 12 is collected for a fixed period (e.g., 10 msec),after which data collection stops and the contents of the rollingbuffers remain fixed for analysis. The stored data may include 3 msec ofdata from before the triggering event. The buffers are then scanned forevery significant peak on the waveform produced by each transducer 12and descriptive parameters such as peak height, peak width, and meanarrival time relative to the initial trigger, are preferably determinedfor each peak.

As shown in FIG. 2, the system 10 may also include a low-frequencytransducer 30 and/or an air transducer 32. The low-frequency transducer30 is used to detect impacts that create low-frequency acoustic signals(e.g., those below approximately 50 hertz). These low-frequency signalsare generally attributed to “bash” impacts on the sensing surface 14,e.g., when a fist bangs a pane of glass. Low-frequency acoustic signals,unlike those created by knuckle taps or metal taps, do not provide astrong response on the preferred piezoelectric transducers.Consequently, the bash-type impacts are not as easily detected with thetransducers 12. Moreover, because bash-type impacts producelow-frequency acoustic signals it is difficult to use the signal toprecisely locate the bash impact, although in some instances anapproximate impact location may be determined. Thus, low-frequencytransducer 30 may detect bash-type impacts in order to inform the userof an error condition if the impact location cannot be localized withsufficient accuracy, or to suggest a different type of impact.Additionally, the very fact of a bash-type impact may have somesignificance in the context of an interactive display (e.g., a game),regardless of its location. In one embodiment, the low-frequencytransducer 30 is an electrodynamic cartridge. In a version of thepreceding embodiment, the diaphragm of the electrodynamic cartridge isattached to the sensing surface 14. Because of the slower response timeof the low-frequency signal detection circuit, an extended samplestorage period is used with the low frequency transducer 30. In oneembodiment, in addition to pre-trigger data storage, 17 msec of posttrigger data is stored for low-frequency impacts.

The air transducer 32 is used as an external noise discriminator, i.e.,to detect ambient noise that can interfere with robust detection. Forexample, sharp sounds (e.g., very loud claps or snaps) produced in thevicinity of the sensing surface 14 are not created by sensing surface 14impacts, but may nonetheless couple enough energy into the system 10 totrigger false impact sensing. In one embodiment, the air transducer 32is a crystal microphone. However, other transducers such as dynamic,piezoelectric, or electret microphones and the like may also be used solong as they respond strongly to ambient sounds that trigger falseimpact sensing. Thus, any events that are detected from this channel ina substantial way (i.e., those having a signal strength above aspecified discriminator threshold) are treated as being produced byexternal sounds, and any resulting signals from transducers 12 areignored by data acquisition system 16. The output signal of airtransducer 32 may be filtered and amplified to eliminate noise andimprove the signal-to-noise ratio of the output signal.

In one embodiment, the signal conditioner 18 includes peak sensingcircuitry that, based on a comparison between the signal strengthreceived from each of transducers 12, 30, and 32, determines if asensing-surface impact or an external sound created the acoustic signal,and if the acoustic signal was created by a surface impact, the type ofimpact (e.g., low frequency) that created the signal. Low-frequencyimpacts and external sounds are generally poorly detected by transducers12. Thus, peak sensing circuitry is advantageous for this purposebecause the peak strength of the transducer 12 output signal iscorrespondingly low for low-frequency impacts and external sounds.

Alternatively, the functions performed by the air transducer 32 mayinstead be performed by additional processing of signals received fromthe transducers 12 in order to discriminate between impacts and externalsounds. Similarly, low-frequency signal detection may be provided byadditional processing of the output signals provided by transducers 12,facilitating omission of the low-frequency transducer 30.

Because some very nearby cellular phone signals may be detected by thetransducers and preamplifiers 12 and interfere with system operation, ameans of preventing false impact sensing resulting from cellular phonesignals can be desirable. In one embodiment, cellular phone interferenceis identified by the relatively rapid signal decay of the cellular phonesignal pulses, and the system does not trigger when such a signal isidentified. In one version of this embodiment, a conventional wave-shapeidentification algorithm in the digital signal processor 20 preventsfalse triggering that would otherwise result from electromagneticinterference.

System Calibration. Because of the unique acoustic characteristics andwide range of available sensing surfaces 14 and impact sources, theimpact-locating system may be calibrated when it is first combined witha sensing surface 14 in order to optimize system performance. In oneembodiment, the calibration is performed by creating surface impacts andrecording the acoustic signal parameters (e.g. signal frequency, signalvelocity, etc.) for each impact. The impacts may be made with impactsources that will be applied to the fully operational system (e.g.,knuckles, metal rings, etc.). The calibration may also include recordingthe dimensions of both the sensing surface 14 and an image-projectionsurface to correlate impact locations with locations within the image.System calibration may be accomplished with a program stored in computer22.

As a result of the calibration, system operation may be improved in thefollowing areas: 1) transducer position optimization; 2) impact sourceidentification accuracy; 3) impact locating accuracy; and 4)correspondence between impacts and interactive display responses. Forexample, where a graphic image is displayed on or adjacent the sensingsurface 14, an accurate translation between the coordinates on thesensing surface 14 and the pixels of the image will enhance systemperformance by facilitating an appropriate response for an impactoccurring at a particular coordinate on the surface 14.

In another example of improvements that may be achieved throughcalibration, transducer 12 locations may be optimized by utilizingmultiple transducer locations during calibration to determine thoselocations which are least affected by dispersion, reflections fromedges, or modal oscillation in a specific sensing surface 14application. In the fully operational system, the transducers 12 may beinstalled at the low-dispersion, low-reflection, and/or nodal locations.

Cross-correlation and Signal Analysis. FIG. 3 illustrates the signalanalysis performed in an embodiment of the present invention. First, instep 36, the input signal 38 is zero-referenced by subtracting anyinitial bias in the signal 38, then normalized, allowing the signals 38from different transducers 12 to be compared regardless of the absoluteamplitude of each signal 38. Then the absolute value can be taken (step40), which eliminates negative signal values that occur due to thegenerally sinusoidal shape of the acoustic signal. Peak matching (step42) is then performed using the normalized signals from a pair oftransducers 12. The objective is to examine the parts of the signal 38that are least corrupted by dispersion or reflections.

Step 36 and step 40 may be performed on the sampled waveform {tilde over(x)}[n] as shown below in Equation (1), which calculates the zero levelby averaging the first 50 samples in the pretrigger (which arrivedbefore the impact wavefront, and hence represent the ambient condition):

$\begin{matrix}{{\overset{\sim}{x}\lbrack n\rbrack} = {{{x\lbrack n\rbrack} - {\frac{1}{50}{\sum\limits_{0 < i \leq 50}{x\lbrack i\rbrack}}}}}} & (1) \\{{\overset{\sim}{x}\lbrack n\rbrack} = \frac{{\overset{\sim}{x}\lbrack {n - 1} \rbrack} + {\overset{\sim}{x}\lbrack n\rbrack}}{2}} & (2)\end{matrix}$To reduce noise in the system, the signal may be filtered using Equation(2). In one embodiment a finite impulse response filter is used.

Because dispersion causes higher frequencies to travel faster, theyarrive first, so it is generally preferred to examine the part of thesignal 38 having the highest dominant frequency. Because all waveformsarise from the same source, certain conditions or heuristics can be usedto identify the optimal peaks or lobes within a signal 38 that will bemostly likely yield meaningful comparisons to corresponding lobes in theother signals 38. For example, in one embodiment, if s_(i) is the samplerepresenting the ith zero-crossing in the sampled waveform {tilde over(x)}[n], the lobes that are chosen from {tilde over (x)}[n] are thosewithin the range s_(i)≦n≦s_(i+N) such that, for some set of constantsδ_(1,) δ_(2,) ε, the following heuristic conditions are met:

$\begin{matrix}{{\max\limits_{S_{i} \leq n \leq S_{i + 1}}{\overset{\sim}{x}\lbrack n\rbrack}} > {{( {1 + \delta_{1}} ){\max\limits_{0 \leq n \leq 50}{\overset{\sim}{x}\lbrack n\rbrack}}} + ɛ}} & (3) \\{{{\max\limits_{S_{i + j + 1} \leq n \leq S_{i + j}}{\overset{\sim}{x}\lbrack n\rbrack}} > {\max\limits_{S_{i + j} \leq n \leq S_{i + j + 1}}{\overset{\sim}{x}\lbrack n\rbrack}}},{1 \leq j \leq {N - 1}}} & (4) \\{{\max\limits_{S_{i + N - 1} \leq n \leq S_{i + N}}{\overset{\sim}{x}\lbrack n\rbrack}} > {\max\limits_{S_{i + N} \leq n \leq S_{i + N + 1}}{\overset{\sim}{x}\lbrack n\rbrack}}} & (5) \\{{\max\limits_{S_{i + N - 2} \leq n \leq S_{i + N - 1}}{\overset{\sim}{x}\lbrack n\rbrack}} < {( {1 + \delta_{2}} ){\max\limits_{S_{i + N - 1} \leq n \leq S_{i + N}}{\overset{\sim}{x}\lbrack n\rbrack}}}} & (6) \\{1 \leq N \leq 4} & (7)\end{matrix}$

Equation (3) ensures that the smallest lobe will be above somethreshhold determined by δ₁ and ε. In one implementation, the valuesδ₁=0.5 and ε=0.1(V) were employed. Because the lobes of interest arethose not “contaminated” with frequencies other than the dominant highfrequency, Equations (4) and (5) are used to select the set of threelargest lobes with ascending maximum heights. It is found, however, thatwhen the waveform saturates quickly, the largest lobe does notnecessarily represent the desired lobe. Hence, Equation (6) is utilizedto ensure that inaccuracies caused by zero-referencing andabsolute-valuing the waveform do not adversely affect the choice oflobes. In particular, setting δ₂=0.1 has been found to address thisproblem adequately. Finally, Equation (7) ensures that at least one lobeis picked, and minimizes calculations by constraining the number oflobes picked to three or fewer.

Having selected the appropriate lobes from each waveform, the next stepis to determine which lobes most closely resemble each other. Assumingconstant attenuation of the signal 38 would require normalization ofeach section before performing any type of comparison. However, todetermine which lobe in the set of three lobes corresponds to the lobesof a set from a different signal 38, it is preferred not to normalizethe entire set to the same height, since any comparison of thesenormalized waveforms would likely lead to the conclusion that the firstlobe in the first set correlates to the first lobe in the second set.

Accordingly, in one implementation, each triplet of lobes is split intotwo pairs: the first and second lobes (lobes 1 and 2), and the secondand third lobes (lobes 2 and 3). Each set is statistically analyzed tofind the distribution of the minimum mean-square error (i.e., thechi-square statistic) of the normalized set of each pair of two lobesfrom two transducers 12 _(a) and 12 _(b):

$\begin{matrix}{{{\chi_{ab}\lbrack n\rbrack} = {\sum\limits_{j = {- {({S_{i + N} - S_{i}})}}}^{S_{i + N} - S_{i}}( {{{\hat{\chi}}_{a}\lbrack j\rbrack} = {{\hat{\chi}}_{b}\lbrack {j - n} \rbrack}} )^{2}}}{where}{{{\hat{x}}_{p}\lbrack n\rbrack} = \{ \begin{matrix}\frac{{\overset{\sim}{\chi}}_{p}\lbrack n\rbrack}{\max_{s_{i} \leq n \leq s_{i + N}}{{\overset{\sim}{\chi}}_{p}\lbrack n\rbrack}} & {s_{i} \leq n \leq s_{i + N}} \\0 & {elsewhere}\end{matrix} }} & (8)\end{matrix}$

The set of two normalized pairs of lobes from transducer 12 _(a) andtransducer 12 _(b) whose minimum chi-square function is smallest, isthen taken and cross-correlated (step 44) to fine-tune the timedifference in samples between the two selected sets of lobes. Theposition of the peak of the cross-correlation function, set forth belowas Equation (9), is used in place of the difference in time-of-flight totransducers 12 _(a) and 12 _(b); in other words, the cross-correlationis used to determine the differential timing of the acoustic signal ateach of the transducers. The cross-correlation is defined conventionallyas:

$\begin{matrix}{{\varphi_{ab}\lbrack m\rbrack} = {\sum\limits_{k = {- \infty}}^{\infty}{{{\hat{x}}_{a}\lbrack k\rbrack}{{\hat{x}}_{b}\lbrack {m - k} \rbrack}}}} & (9)\end{matrix}$where m and k are indices. In particular, the cross-correlationoperation between two signals creates a new signal, indexed as m, andsummed over all possible values k from −∞ to +∞. In one embodiment,cross-correlation is improved by locating transducers 12 adjacent oneanother. The improved cross-correlation results because adjacenttransducers 12 may detect a similar acoustic signal as a result of theirproximity.

It is to be noted that not all of the calculations described above needbe performed to find the peak of the cross-correlation. In particular,it is not necessary to calculate every chi-square statistic, only tothrow away all but one of them, i.e., the minimum chi-square statistic.Thus, it is possible to calculate the chi-square statistic for a rangeof n where it is likely the minimum will occur. In general, a sufficientrange is n=χ±10 where χ is the average of the “center” of the two setsof lobes. Furthermore, since {circumflex over (χ)}_(a)[k] and{circumflex over (χ)}_(b)[k] are finite signals, the cross-correlationis straightforwardly computed.

Other suitable approaches to removing or compensating for the effects ofdispersion include iterative signal processing and similar compensationtechniques. For example, an adaptive filter can be used in conjunctionwith the localizing algorithm to “stiffen” the rising edge by removingdispersion, thereby yielding a better impact estimate; iterating thisprocess results in a better signal representation and more accurateimpact localization. In a further approach, the acoustic signal positionmay be tracked based on a linear prediction of the signal position.Compensation for dispersion is accomplished by iterating the predictionprocess over multiple signal positions. The predictive approach may beimplemented with filters such as adaptive filters, Kalman filters,Wiener filters, and the like provided that the filters solve for aminimum mean square error. In another approach, iterative calculationsare made based on estimated velocity of various frequency components ofthe acoustic signal. The process includes adjusting the velocityestimates, and calculating both the time of arrival of the signal ateach transducer 12 and the corresponding impact coordinates. Thedifferential time of arrival of the acoustic signal at transducer 12pairs may be determined by either a rising-edge method or the signalanalysis shown in FIG. 3. A heuristic may be used to adjust the arrivaltime difference between the signal arrival at each transducer until theimpact coordinate calculated for each transducer pair is substantiallythe same and the difference between velocity vectors is minimized. Inone embodiment, the impact coordinate is determined using Equation (10).

In still a further approach, an iterative method is applied to theacoustic signal in the frequency domain. Here, the effects of signaldispersion are reversed and the acoustic signal is compressed to asingle pulse with a distinct rising edge and associated peak amplitudeby phase-shifting the signal's dominant frequencies. Iterativecalculations are performed to determine when the compensation applied toeach frequency component of the signal is correct. When correctcompensation occurs the compensated signal matches an ideal pulse andwill pass a matched filter output threshold. The matched filter mayinclude an ideal pulse compression. The immediately preceding approachmay be simplified by modeling the waveform as the sum of itsphase-shifted dominant frequency components. In one embodiment, themodel is created based on data produced during a system calibration.Signal compression may also be achieved without the preceding signalanalysis by applying transducers 12 that perform signal compression.

The difference in arrival times computed as set forth above are thenused to calculate the impact location P. For example, let t_(ab) be thetime difference, in samples, between the arrival time at firsttransducer 12 _(a) and the arrival time at second transducer 12 _(b).The impact location P is determined using the following equation:

$\begin{matrix}{\frac{t_{ab} \times c}{{sample}\mspace{14mu}{rate}} = {\sqrt{( {x - x_{a}} )^{2} + ( {y - y_{a}} )^{2}} - \sqrt{( {x - x_{b}} )^{2} + ( {y - y_{b}} )^{2}}}} & (10)\end{matrix}$

In Equation (10), x and y are the coordinates of the impact location onthe surface. The first transducer 12 _(a) is located at coordinatesx_(a) and y_(a,) and the second transducer 12 _(b) is located atcoordinates x_(b,) and y_(b). The acoustic signal transmission speed isrepresented by c. Thus, the speed of the acoustic signal is necessary todetermine the impact location.

Where the acoustic signal transmission speed varies depending upon theimpact source, the impact-locating system identifies the impact sourcein order to solve Equation (10) and identify the impact location P.

In one implementation, the impact source is identified by a nearestneighbor algorithm set forth below as Equation (11). The algorithmincludes six variables that may be used to characterize the impactsource. Four variables z_(1,) z_(2,) z_(3,) and z₄ provide an estimateof the frequency content of the acoustic signal received at each of fourrespective transducers 12 _(1,) 12 _(2,) 12 _(3,) and 12 ₄. Twovariables, c₁ and c_(2,) each provide information related to thevelocity of the acoustic signal. The values of the six variablesresulting from a new impact are compared with known values of thevariables in order to identify the unknown impact source. The knownvalues are calibration samples previously recorded during a surfacecalibration with one or more known impact sources.

$\begin{matrix}{{d\lbrack i\rbrack} = \sqrt{{\sum\limits_{j = 1}^{4}\frac{( {z_{j:k} - z_{j:i}} )^{2}}{m}} + {\sum\limits_{j = 1}^{2}\frac{( {c_{j:k} - c_{j:i}} )^{2}}{n}}}} & (11)\end{matrix}$

In one approach, the nearest neighbor algorithm determines if the impactsource is either a knuckle tap or a metal tap. The variables z_(1,)z_(2,) z_(3,) and z₄ provide the total quantity of acoustic signal zerocrossings detected by the transducer 12 associated with the variable(e.g., z₁ is the number of zero crossings detected at transducer 12 ₁)during a given period. In one example, the quantity of zero crossingsdetected over a 400-sample range is used. The time difference, measuredin samples, between the receipt of the signal at transducer 12 ₁ and thereceipt of the signal at transducer 12 ₃ is represented by c₁. The timedifference, measured in samples, between the receipt of the signal attransducer 12 ₂ and the receipt of the signal at transducer 12 ₄ isrepresented by c₂. The range of values for z_(1,) z_(2,) z_(3,) and z₄is [0, m], and the range of values for c₁ and c₂ is [0, n].

The calibration sample set includes multiple values for each impactsource, for example, data from 100 knuckle taps and 100 metal taps. Thisdata may be represented as follows:t[i]=z_(1:i,) z_(2:i,) z_(3:i,) z_(4:i,) c_(1:i,) c_(2:I)where i is the sample number for each of the calibration samples. Thecalibration samples are organized so that samples from each type ofimpact source are grouped together. For example, the data from the onehundred knuckle taps may be represented by samples i=1 to 100, and thedata from the one hundred metal taps may be represented by samples i=101to 200.

For a newly-occurring surface impact with the following characteristics:t[k]=z_(1:k,) z_(2:k,) z_(3:k,) z_(4:k,) c_(1:k,) c_(2:k)the impact source is determined with Equation (11) by identifying thefive calibration samples i such that d[i] has the smallest value overthe entire set of samples i. A simple majority of the five samples iscalculated to determine if among the five identified samples themajority of samples were found within either the knuckle tap samplegroup (i=1 to 100) or the metal tap sample group (i=101 to 200). Ifamong the majority i>100 the impact source is identified as a metal tap.Conversely, if among the majority i<100 the impact source is identifiedas a knuckle tap. Thus, the difference in signal arrival time at thetransducers and the number of zero crossings in the signal arecorrelated to determine the impact source.

The number of calibration samples is selected to avoid incorrect sourceidentification. Too few calibration samples may lead to inaccuraciesbecause too few acoustic signatures are represented. Conversely, toomany calibration samples may make the approach too restrictive. Ofcourse, the preceding approach is not limited to knuckle tap and metaltap identification, and may be used with any number of impact sourceshaving signal characteristics that can be distinguished from oneanother. Where more than two impact sources exist, the number of samplegroups will increase to match the number of impact source classes (e.g.,three sample groups are provided when the impact source may beclassified as being a knuckle tap, a metal tap, or a wood tap).

Not all types of impacts need be analyzed in the same fashion. It isfound, for example, that while a combination of correlation and risingedge results may is useful in detecting soft impacts such as knuckletaps and determining the impact location, for hard impacts such as metaltaps, the leading edge of the acoustic signal can frequently bedetermined, obviating the need for correlation and rising edge results.Further, acoustic signal arrival time may be determined by a identifyingthe signal's initial rising edge and then locating the zero-crossingthat occurs immediately following the rising edge. In one embodiment,zero-crossing detection includes narrowing the signal's bandwidth with abandpass filter.

The impact location is then determined based on the intersection of thehyperbolas defined by Equation (10) as applied to all transducers 12. Acalibration may be performed to tailor the computation to the sensingsurface 14. The calibration provides a greater degree accuracy forimpact-location calculations because propagation characteristics mayvary within a surface even when the surface material is contiguous. Forexample, mechanical loading may affect the propagation velocity of anacoustic signal traveling in one region of the surface.

It is found that metal-tap impact-location results tend to be lessaccurate than results obtained for soft impacts because of the higherpropagation velocity of the acoustic signal generated by hard impactsand ambiguities in determining the rising edge. Generally, higher signalpropagation velocities found in thicker material result in decreasedimpact location accuracy. In practice, the impact-locating systemdescribed herein will detect impacts located outside the region boundedby the transducers 12. Accuracy in these regions, especially in thedirection orthogonal to the line between the sensors, is generally isnot as high as position locating within the bounded region (i.e., theregion located between the transducers) because of the asymptotic changeof the differential time. Nonetheless, because the overall accuracy oflocalization based on hyperbolas depends on the sampling rate of digitalsignal processor 20 (since greater sampling leads to more accuratedefinitions of peaks and zero-crossings), improvement can be obtained byincreasing this rate. Other techniques such as adaptive filtering canalso be used to advantage.

If it is assumed, for a rectangular sensing surface 14 with a firstdimension a and a second dimension b (where a>b), that the attenuationis proportional to the distance that the signal travels from the impactorigin to transducers 12, then the maximum sensing area defined by thebounded region created by transducers 12 mounted at the sensing surface14 corners is:b√{square root over (x²−b²)}  (12)where x is the maximum distance between the impact origin and thetransducers 12 before the acoustic signal becomes too attenuated.

In comparison, the maximum sensing area defined by the bounded regioncreated by transducers 12 mounted at the center of each side of thebounded region is:π×xb   (13)

The system provides a rapid response time, making it suitable forinteractive applications. For example, the digital signal processor 20generally produces parameters within 50 msec of surface impact. In oneembodiment, the computer 22 produces the impact coordinates 15 msecafter receiving the parameters from the digital signal processor 20. Thepreceding combination of processing and calculating times yields a netsystem response time of approximately 65 msec. In a further embodiment,the cost is reduced and the response time of the system is potentiallyimproved by replacing the digital signal processor 20 with afour-channel audio digitizer and performing all processing withincomputer 22.

As discussed previously, the rapid system-response time makes theimpact-locating system advantageous for interactive displays. Referringagain to FIG. 2, in one embodiment, the interactive content is storedand/or generated by in the host computer 22 in a conventional fashion.For example, an impact-locating system in accordance with the presentinvention can be used on a glass storefront with a graphic imageprojected onto a screen behind the glass. A user may then interact withthe displayed information by knocking, tapping, and banging on thesensing surface 14. The user's impacts on the surface can be used tochange the image contents and the image location on the sensing surface14. Further, the nature of the impact (e.g., bash, knuckle tap, sharpimpact, etc.) may also be used to vary the image displayed to the user.Additionally, a speaker may provide audible feedback (e.g., voiceinstructions, musical notes, etc.) to a user where the nature of theaudible feedback varies as a result of the impact source, impactlocation, or both. Today, most individuals are very familiar withcomputer graphical interfaces wherein the user clicks images or clicks,holds, and drags images on the computer screen. If taps are analogizedto clicks, then the addition of a sensor, such as a laser rangefinder orvideo tracker, for determining lateral movement adjacent the sensingsurface 14 completes a graphical user interface thereon, with lateralmovements analogized to dragging an image across the display area. Inanother embodiment, the location of objects in the vicinity of thesensing surface 14 is determined with a low-power range-finding radar.Feedback provided by rangefinders may also be used to activate agraphical interface from a “sleep mode” when a user approaches thesensing surface 14, e.g., to limit surface impact analysis only toperiods when a user is proximate to the sensing surface 14. Again,bash-type impacts may be sensed for informational significance or totrigger a warning of inappropriate interaction with the sensing surface14.

In a further embodiment, the system is retrofitted to an existingsurface to create a sensate surface from an inanimate surface. Theretrofit is accomplished by mounting the transducers 12 adjacent thesensing surface 14 and connecting the transducers 12, 30, 32 to the dataacquisition system 16. The systems and methods of the present inventionmay be used to track surface impacts on sensing surfaces 14 fabricatedfrom a wide variety of materials such as glass, metal, metal alloys,plastic, composites, laminates, and the like so long as the materialwill transmit an acoustic signal. Additionally, the systems and methodsof the invention may be used regardless of the sensing surfaceorientation. In one embodiment, the invention is used with a horizontalsurface, for example, a ping-pong table. In another embodiment, theinvention is used with a vertical surface, for example, a glassstorefront window.

While a preferred embodiment is disclosed, many other implementationswill occur to one of ordinary skill in the art and are all within thescope of the invention. Each of the various embodiments described abovemay be combined with other described embodiments in order to providemultiple features. Furthermore, while the foregoing describes a numberof separate embodiments of the apparatus and method of the presentinvention, what has been described herein is merely illustrative of theapplication of the principles of the present invention. Otherarrangements, methods, modifications, and substitutions by one ofordinary skill in the art are therefore also considered to be within thescope of the present invention, which is not to be limited except by theclaims that follow.

1. A method of locating an impact on a surface, the method comprisingthe steps of: receiving, with a plurality of sensors located at aplurality of locations on the surface, an acoustic signal created by theimpact; with at least one data processing system, performing the stepsof: determining a time of arrival of the acoustic signal at each of theplurality of locations by cross-correlation of the acoustic signalreceived at the locations; identifying an impact source for the receivedacoustic signal using at least one of a correlation of the acousticsignal received at the locations, a rising edge analysis, a nearestneighbor analysis, or a leading edge analysis; and based on thedetermined time of arrival and the identity of the impact source,determining the impact location.
 2. The method of claim 1 wherein theacoustic signal includes flexural waves.
 3. A method of locating animpact on a surface, the method comprising the steps of: receiving, witha plurality of sensors located at a plurality of locations on thesurface, an acoustic signal created by the impact; with at least onedata processing system, performing the steps of: ignoring acousticsignals created by external sounds; identifying an impact source for thereceived acoustic signal using at least one of a correlation of theacoustic signal received at the locations, a rising edge analysis, anearest neighbor analysis, or a leading edge analysis; and based on theacoustic signal created by the impact and the identity of the impactsource and not on acoustic signals created by external sounds,calculating an impact location.
 4. The method of claim 3 furthercomprising the step of using air transducer to detect acoustic signalscreated by external sounds.
 5. The method of claim 4 wherein the airtransducer is a crystal microphone.
 6. A method of locating an impact ona surface, the method comprising the steps of: receiving, with aplurality of sensors located at a plurality of locations on the surface,an acoustic signal created by the impact; with at least one dataprocessing system, performing the steps of: detecting at one or morelocations a low-frequency acoustic signal created by the impact;identifying an impact source for the received acoustic signal using atleast one of a correlation of the acoustic signal received at thelocations, a rising edge analysis, a nearest neighbor analysis, or aleading edge analysis; and based on the acoustic signal received at thelocations, the identity of the impact source, and detection of thelow-frequency signal, calculating an impact location.
 7. The method ofclaim 6 wherein the low frequency is below 50 Hertz.
 8. The method ofclaim 6 wherein the low frequency wave is detected by an electrodynamiccartridge.
 9. A method of locating an impact on a surface, the methodcomprising the steps of: receiving, with a plurality of sensors locatedat a plurality of locations on the surface, an acoustic signal createdby the impact; with at least one data processing system, performing thesteps of: identifying a wavefront of the acoustic signal after removingwave dispersion effects; identifying an impact source for the receivedacoustic signal using at least one of a correlation of the acousticsignal received at the locations, a rising edge analysis, a nearestneighbor analysis, or a leading edge analysis; and based on a time ofarrival of the wavefront at the locations and the identity of the impactsource, calculating an impact location.
 10. The method of claim 9further comprising the step of creating an acoustical model of thesurface.
 11. The method of claim 9 further comprising the step ofidentifying an impact source.
 12. A system for locating an impact on asurface comprising: a plurality of transducers associated with thesurface to detect an acoustic signal created by the impact; and aprocessor, responsive to the transducers, for identifying an impactsource for the received acoustic signal using at least one of acorrelation of the acoustic signal received at the locations, a risingedge analysis, a nearest neighbor analysis, or a leading edge analysisand for determining an impact location based on a cross-correlation ofthe acoustic signal as detected by the transducers and the identity ofthe impact source.
 13. The system as claimed in claim 12 wherein thecross-correlation facilitates determination of a time of arrival of theacoustic signal at each of the transducers.
 14. A system for creating agraphical user interface on a surface, the system comprising: aplurality of sensors for detecting surface impacts; a data-acquisitionsystem for identifying an impact source for the received acoustic signalusing at least one of a correlation of the acoustic signal received atthe locations, a rising edge analysis, a nearest neighbor analysis, or aleading edge analysis and for determining impact locations on thesurface using cross-correlation and the identity of the impact source;and an image generator, responsive to the data-acquisition system, fordisplaying a graphical image on the surface based on the impactlocations.
 15. The system as claimed in claim 14 further comprising alaser rangefinder.
 16. A method of characterizing an impact source, themethod comprising the steps of: calibrating a data-acquisition systemwith known acoustic-signal characteristics for at least one impactsource; receiving, with a plurality of sensors located at a plurality oflocations on the surface, an acoustic signal created by the impactsource to be characterized; analyzing the received acoustic signal withthe data-acquisition system using at least one of a correlation of theacoustic signal received at the locations, a rising edge analysis, anearest neighbor analysis, or a leading edge analysis; and basedthereon, characterizing the impact source or type of impact.
 17. Themethod as claimed in claim 16 further comprising the step of determininga wave velocity.